{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Cost Function for Logistic Regression\n",
    "\n",
    "In this lab, you will implement the cost function for logistic regression.\n",
    "\n",
    "## Dataset \n",
    "Let's start with the same dataset as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = np.array([[0.5, 1.5], [1,1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])\n",
    "y = np.array([0, 0, 0, 1, 1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use a helper function to plot this data. The data points with label $y=1$ are shown as red crosses, while the data points with label $y=0$ are shown as black circles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lab_utils import plot_data\n",
    "\n",
    "plot_data(X,y)\n",
    "\n",
    "# Set both axes to be from 0-6\n",
    "plt.axis([0, 6, 0, 6])\n",
    "# Set the y-axis label\n",
    "plt.ylabel('$x_2$')\n",
    "# Set the x-axis label\n",
    "plt.xlabel('$x_1$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost function\n",
    "\n",
    "First, you will implement a non-vectorized version of the cost function. Then, you will implement a vectorized version.\n",
    "\n",
    "\n",
    "### Non- vectorized version\n",
    "\n",
    "Recall that for logistic regression, the cost function is of the form \n",
    "\n",
    "$$ J(w) = \\frac{1}{m} \\sum_{i=1}^{m} \\left[ cost(f_{w}(x^{(i)}), y^{(i)}) \\right]$$\n",
    "\n",
    "where\n",
    "* m is the number of training examples in the dataset\n",
    "\n",
    "* $cost(f_{w}(x^{(i)}), y^{(i)})$ is the cost for a single data point, which is - \n",
    "\n",
    "    $$cost(f_{w}(x^{(i)}), y^{(i)}) = (-y^{(i)} \\log\\left(f_w\\left( x^{(i)} \\right) \\right) - \\left( 1 - y^{(i)}\\right) \\log \\left( 1 - f_w\\left( x^{(i)} \\right) \\right)$$\n",
    "    \n",
    "*  $f_{w}(x^{(i)})$ is the model's prediction, while $y^{(i)}$, which is the actual label\n",
    "\n",
    "\n",
    "* For a logistic regression model for the dataset given above, the model can be representented as:\n",
    "\n",
    "    $f_{w}(x) = g(w_0 + w_1x_1 + w_2x_2)$\n",
    "\n",
    "    where $g(z)$ is the sigmoid function:\n",
    "\n",
    "    $g(z) = \\frac{1}{1+e^{-z}}$ \n",
    "\n",
    "\n",
    "* **Preprocessing step** \n",
    "\n",
    "   For ease of implementation, we will add an additional column of ones to $X$ (as $x_0$) so that  \n",
    "    $f_{w}(x) = g(w_0x_0 + w_1x_1 + w_2x_2)$\n",
    "    \n",
    "    By doing this, to calculate the prediction from the model $f_{w}(x)$, we can write a for loop that calculates $w_jx_j$ at each step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a column to X_orig to account for the w_0 term\n",
    "X_mod = np.hstack([np.ones((X.shape[0],1)), X])\n",
    "\n",
    "print(X_mod)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Side Note: sigmoid function implementation\n",
    "We've implemented the `sigmoid` function for you already and you can simply import and use it, as shown in the code block below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lab_utils import sigmoid \n",
    "\n",
    "print(sigmoid(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Implementation\n",
    "\n",
    "Now, you'll implement the non-vectorized version of the cost function. We've already provided some starter code for you which does the following -\n",
    "* There is a for loop over all the examples in the dataset\n",
    "* There is a list to store the cost calculated for each example that we add to \n",
    "* The total cost is then computed as the sum of cost for each example divided by the number of examples\n",
    "\n",
    "We assume that the function takes in the paramaters $w$ as a list/array.\n",
    "\n",
    "\n",
    "**Exercise**\n",
    "\n",
    "You'll complete the cost function by implementing the following steps inside the for loop - \n",
    "\n",
    "* First, you'll compute the models prediction $f(x^{(i)})$ for a single data point at index $i$ as shown below.\n",
    "\n",
    "   ```\n",
    "   z = 0\n",
    "   for j in range(n):\n",
    "       z += w[j] * X[i][j]\n",
    "   f = sigmoid(z)\n",
    "   ```\n",
    "   \n",
    "   Since $w_0x_0 + w_1x_1+w_2x_2 = w^T\\cdot x$, you can also calculate  $f(x^{(i)})$ as \n",
    "   ```\n",
    "   z = np.dot(w.T, X[i])\n",
    "   f = sigmoid(z)\n",
    "   ```\n",
    "\n",
    "* Then, you'll compute the cost for the single data point at index $i$ as \n",
    "\n",
    "  ```\n",
    "  cost = -y[i]*np.log(f) - (1-y[i])*np.log(1-f)\n",
    "  ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(X, y, w):\n",
    "    # Here X is assumed to pre-processed with a column of ones added as x_0\n",
    "\n",
    "    m, n = X.shape\n",
    "    cost_list = []\n",
    "    \n",
    "    for i in range(m):\n",
    "        ### START CODE HERE ### \n",
    "        \n",
    "        ### END CODE HERE ### \n",
    "        \n",
    "        cost_list.append(cost)\n",
    "        \n",
    "    total_cost = (1/m)* sum(cost_list)\n",
    "    return total_cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the implementation of your cost function using the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = np.array([-3,1,1])\n",
    "print(compute_cost(X_mod, y, w1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**: 0.3668667864055174"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see what the cost function output is for a different value of $w$. \n",
    "\n",
    "* In the last ungraded lab, you plotted the decision boundary for  $w_0 = -3, w_1 = 1, w_2 = 1$. That is, you had `w = np.array([-3,1,1])`.\n",
    "\n",
    "* Let's say you want to see if $w_0 = -4, w_1 = 1, w_2 = 1$, or `w = np.array([-4,1,1])` provides a better model.\n",
    "\n",
    "Let's first plot the decision boundary for these two different $w_0$ values to see which one fits the data better.\n",
    "\n",
    "* For $w_0 = -3, w_1 = 1, w_2 = 1$, we'll plot $-3 + x_1+x_2 = 0$, which is equivalent to $x_2 = 3 - x1$ (shown in blue)\n",
    "* For $w_0 = -4, w_1 = 1, w_2 = 1$, we'll plot $-4 + x_1+x_2 = 0$, which is equivalent to $x_2 = 4 - x1$ (shown in magenta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Choose values between 0 and 6\n",
    "x1 = np.arange(0,6)\n",
    "\n",
    "# Plot the two decision boundaries\n",
    "x2 = 3 - x1\n",
    "x2_other = 4 - x1\n",
    "\n",
    "# Plot the decision boundary\n",
    "plt.plot(x1,x2, c=\"b\", label=\"$w_0$=-3\")\n",
    "plt.plot(x1,x2_other, c=\"m\", label=\"$w_0$=-4\")\n",
    "plt.axis([0, 6, 0, 6])\n",
    "\n",
    "# Plot the original data\n",
    "plot_data(X,y)\n",
    "# Set the y-axis label\n",
    "plt.ylabel('x2')\n",
    "# Set the x-axis label\n",
    "plt.xlabel('x1')\n",
    "\n",
    "plt.legend(loc=\"upper right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see from this plot that `w = np.array([-4,1,1])` is a worse model for our data. Let's see if you can see this from your cost function implementation as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w_array1 = np.array([-3,1,1])\n",
    "w_array2 = np.array([-4,1,1])\n",
    "\n",
    "print(\"Cost for w0 = -3 : \", compute_cost(X_mod, y, w_array1))\n",
    "print(\"Cost for w0 = -4 : \",compute_cost(X_mod, y, w_array2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**\n",
    "\n",
    "Cost for w0 = -3 :  0.3668667864055174\n",
    "\n",
    "Cost for w0 = -4 :  0.503680863674846\n",
    "\n",
    "\n",
    "You can see that your cost function behaves as expected and the cost for `w = np.array([-4,1,1])` is indeed higher than the cost for `w = np.array([-3,1,1])`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Optional ) Vectorized version\n",
    "\n",
    "You will now implement a vectorized version of the cost function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vectorized version of the cost function formula is \n",
    "\n",
    "$$ J(w) = \\frac{1}{m} (-y^T \\cdot \\text{log}(f) - (1-y)^T \\cdot \\text{log}(1-f))$$ \n",
    "\n",
    "where\n",
    "\n",
    "$$ f = g(X \\cdot w)$$\n",
    "\n",
    "As before, $g$ is the sigmoid function\n",
    "\n",
    "**Exercise**\n",
    "\n",
    "You'll complete the vectorized cost function by implementing the following steps - \n",
    "\n",
    "* First, you'll compute the models prediction $f(x)$ as shown below\n",
    "\n",
    "   ```\n",
    "   z = np.dot(X, w)\n",
    "   f = sigmoid(z)\n",
    "   ```\n",
    "  \n",
    "\n",
    "* Then, you'll compute the cost as \n",
    "\n",
    "  ```\n",
    "  total_cost = (1/m)*(np.dot(-y.T, np.log(f)) - np.dot((1-y).T, np.log(1-f)))\n",
    "  ```\n",
    "\n",
    "**Debugging Tip:** Vectorizing code can sometimes be tricky. One common strategy for debugging is to print out the sizes of the matrices you are working with using the size function. For example, given a data matrix $X$ of size 6 × 3 (6 examples, 3 features) and $w$, a vector with dimensions 3x1, you can observe that $Xw$ is a valid multiplication operation, while $wX$ is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost_vectorized(X, y, w):\n",
    "    # Here X is assumed to pre-processed with a column of ones added as x_0\n",
    "    m = X.shape[0]\n",
    "    \n",
    "    ### START CODE HERE ### \n",
    "\n",
    "    ### END CODE HERE ### \n",
    "    \n",
    "    return total_cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check if the output of this function is equivalent to the output of your non-vectorized implementation above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Cost computed by non-vectorized version: \", compute_cost(X_mod, y, w))\n",
    "print(\"Cost computed by vectorized version: \", compute_cost_vectorized(X_mod, y, w))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected output**\n",
    "\n",
    "Cost computed by non-vectorized version:  0.3668667864055174\n",
    "\n",
    "Cost computed by vectorized version:  0.3668667864055174\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
